On-line 3D Body Modelling for Augmented Reality

Building 3D body models is an important task for virtual and augmented reality applications in telerehabilitation, education, 3DTV, entertainment and tele-presence. We propose a real-time full 3D reconstruction system that combines visual features and shape-based alignment using low cost depth sensor and video cameras targeting three-dimensional conferencing applications. With this approach we overcome the classic video based reconstruction problem in low-texture or repeated pattern regions. Alignment between successive frames is computed by jointly optimizing over both appearances and shape matching. Appearance-based alignment is done over 2D SURF features annotated with 3D position. Shape-based alignment is performed using the motion transformation estimation between consecutive annotated 3D point clouds through a linear method. A solution to avoid wrong annotated 3D matched points is proposed. 3D mesh model representation is used to lower the processed data and create a 3D representation that is independent of view-point.